Method and system for visualizing a volume dataset

ABSTRACT

A machine-implemented display method that, with respect to a volume dataset being rendered, enables a user to navigate to any position in space and look in any direction. Preferably, the volume dataset is derived from a computer tomography (CT) or magnetic resonance imaging (MRI) scan. With the described approach, the user can see details within the dataset that are not available using conventional visualization approaches. The freedom-of-motion capability allows the user to go to places (positions) within the volume rendering that are not otherwise possible using conventional “orbit” and “zoom” display techniques. Thus, for example, using the described approach, the display image enables a user to travel inside physical structures (e.g., a patient&#39;s heart, brain, arteries, and the like).

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates generally to medical imaging.

2. Background of the Related Art

Medical imaging is the technique used to create images of the human bodyor parts thereof for clinical purposes (medical procedures that seek toreveal, to diagnose or to examine disease) or medical science (includingthe study of normal anatomy and physiology). Computer tomography (CT)and magnetic resonance imaging (MRI) are two of the most commonapproaches. These techniques generate a set of individual 2D images thatcan be displayed in a 3D visualization as a “volume dataset.” Typically,however, the extent of the 3D visualization is limited to “orbiting” and“zooming.” In an “orbit” mode, the view of the object being rendered islike an orbiting satellite in that the viewer can move around the objectbeing viewed from any angle but cannot look “out” from a position withinthe object. A zoom operation provides the viewer with additional usefuldetails about the object; however, zooming does not enable the viewer tomove down to a surface or inside of a volume. Thus, the orbit and zoomapproach has limited applicability for rendering and viewing a volumemedical dataset.

BRIEF SUMMARY

The disclosed subject matter provides a machine-implemented displaymethod that, with respect to a volume dataset being rendered, enables auser to navigate to any position in space and look in any direction.Preferably, the volume dataset is derived from a computer tomography ormagnetic resonance imaging scan. With the described approach, the usercan see details within the dataset that are not available usingconventional visualization approaches. The freedom-of-motion capabilityallows the user to go to places (positions) within the volume renderingthat are not otherwise possible using conventional “orbit” and “zoom”display techniques. Thus, for example, using the described approach, thedisplay image enables a user to travel inside physical structures (e.g.,a patient's heart, brain, arteries, and the like).

In one embodiment, a rendering method is implemented on a machine, suchas a computer that includes a display. The machine receives a volumedataset generated by the CT or MRI scan. Typically, the dataset is a setof digital data comprising a set of individual 2D images. An image ofthe volume dataset is rendered on the display at a given number offrames per second, where each frame of the image has pixels that areuniform. According to the technique, any frame (within a set of framesbeing displayed at the display rate) may have a resolution that differsfrom that of another frame. This concept is referred to herein ascontinuous real-time dynamic rendering resolution. Moreover, any pixelwithin a given frame may intersect a ray cast forward into the view at apoint that differs from that of another pixel in the frame. This latterconcept is referred to herein as continuous per pixel dynamic samplingdistance for ray tracing within the volume dataset. Thus, according to arendering method herein, at least two frames of an image sequence havevarying resolution relative to one another, and at least two pixelswithin a particular frame have a varying number of ray tracing stepsrelative to one another. When the volume dataset is rendered in thismanner, a viewer can navigate to any position and orientation within the3D visualization.

According to another aspect of this disclosure, the machine provides theuser with a “virtual camera” that the user can control with an inputdevice (e.g., a pointing device, keyboard, or the like) to facilitaterendering on a display monitor of a display object from any point inspace and in any direction at real-time display frame update rates. Asnoted above, preferably the volume dataset is rendered using bothdynamic rendering resolution and per pixel dynamic sampling distance forray tracing. This rendering approach enables the user to move and rotatethe virtual camera in response to the rendered image from any pointwithin or outside the image.

The foregoing has outlined some of the more pertinent features of theinvention. These features should be construed to be merely illustrative.Many other beneficial results can be attained by applying the disclosedinvention in a different manner or by modifying the invention as will bedescribed.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention and theadvantages thereof, reference is now made to the following descriptionstaken in conjunction with the accompanying drawings, in which:

FIG. 1 illustrates a computer system coupled to a medical imagingsystem;

FIG. 2 illustrates a known technique for rendering and “orbiting” abouta volume dataset;

FIG. 3 illustrates how ray tracing is used to generate a final imageusing a virtual camera;

FIG. 4 illustrates the technique of this disclosure for rendering andviewing a volume dataset from any position external to or within thedataset;

FIG. 5 illustrates a dynamic rendering resolution technique of thisdisclosure;

FIG. 6 illustrates a known fixed step ray tracing technique;

FIG. 7 illustrates a dynamic step ray tracing technique of thisdisclosure;

FIG. 8 illustrates the dynamic stepping approach showing how the numberof steps varies along the ray in areas where tissues are located;

FIG. 9 is another view showing the dynamic stepping approach of FIG. 8;and

FIG. 10 is a machine in which the disclosed visualization methods may beimplemented.

DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT

As illustrated in FIG. 1, a system 100 in which the subject matterherein is implemented comprises a computer system 102 having a displaymonitor 108, and one or more input devices 110 such as a keyboard, apointing device, or the like. The computer system 102 is illustrated asa desktop workstation, but this is not a limitation, as the system maybe implemented in a laptop or notebook computer, a wireless computingdevice (such as an iPad), a mobile handheld device (including a smartphone with application support), or any other computing machine thatincludes a display. The techniques of this disclosure are not limited toany particular type of computing device, system or architecture, and oneor more of the elements of the machine may be located in differentlocations. Thus, for example, the display monitor may be positionedremotely from other components. For convenience of illustration only,the computer system 102 is shown as receiving inputs from a pair ofimaging devices 106 that are associated with a support 104. The support104 rotates or reciprocates relative to the imaging devices 106 togenerate a set of individual 2D images of an object being scanned.Typically, the support 104 has associated mechanical elements, hydraulicelements and/or electronic elements (not shown) that control theposition or rotational speed thereof. The support may be under computercontrol. Likewise, the one or more imaging devices 106 includeassociated optical elements, mechanical elements, and/or other controlelements that control the position and operation of the device.Typically, an object to be imaged (e.g., a human body, or some partthereof) is located on the support 104. The support may be fixed, inwhich case the imaging devices 106 rotate or reciprocate with respectthereto. One of ordinary skill in the art will appreciate that thesupport 104 and imaging devices 106 may represent conventional medicalimaging systems such as computer tomography (CT), magnetic resonanceimaging (MRI), or the like. The techniques herein may be used to producea 3D visualization of other digital data, irrespective of how thedataset is generated. Typically, such imaging systems are external tothe display system and method of this disclosure, although thetechniques herein may be implemented natively within such known imagingsystems. The 2D images comprising a particular scan typically conform toa standard digital data format (e.g., DICOM) and are received by thecomputer system 102 in any convenient manner, e.g., a CD, DVD, USBstick, hard drive, network drive, PACS (a medical CT library), or thelike. The computer system 102 may be network-accessible, in which casethe digital data comprising the volume dataset may be received over acommunication network, such as any Internet Protocol (IP)-based network,a wireline network, a wireless network, a combination thereof, or thelike. As noted above, this disclosure provides a display method,preferably implemented in a computer, such as a workstation as shown inFIG. 1. More generally, the method is implemented using one or morecomputing-related entities (systems, machines, processes, programs,libraries, functions, code, or the like) that facilitate or provide theinventive functionality. In a representative but non-limitingimplementation, the display methods described herein are implemented ina machine comprising a CPU (central processing unit), such as any Intel-or AMD-based chip, computer memory, such as RAM (at least 1 GB), a harddrive (at least 8 GB), and a CD-drive (preferably 24-48×). The machinesoftware includes an operating system (e.g., Windows XP, Windows Vista,Windows 7, any Apple OS, either 32 bit or 64 bit), and generic supportapplications. If the process is implemented in a graphics processor,preferably the machine comprises a graphics processing unit (GPU) suchas the AMD Radeon Series 4850 or equivalent (preferably at least DirectX9-compliant and Pixel Shader 3.0-compliant).

By way of background, FIG. 2 illustrates a known technique forvisualizing a volume dataset 200. The volume dataset is a stack orassembly of 2D images forming a cube (or 3D dataset). In this approach,a virtual camera 202 orbits around the dataset along the constraint 204,which constraint is shown in the drawing as a circle but is actually asphere in three dimensions. The virtual camera 202 is amachine-implemented software-generated construct that has a position, anorientation, and a resolution. The virtual camera renders a final image206 at a display rate by ray tracing. In the drawing, a single virtualcamera is shown at various positions along the orbital constraint. Ineach position, the virtual camera produces a final image in the mannerillustrated in FIG. 3. Ray tracing is a process by which a set of raysare cast forward into the volume dataset from the virtual camera andintersected with structures therein to produce the final imagecomprising a set of pixels. In particular, a ray 208, which is a linewith a direction located at a point in space, is cast forward into thescene to find an intersection point or multiple intersection points thatcontribute to the final value (brightness or color) for a single pixel210 in the final image. Each “ray” is generated in software andsimulates light generated from the camera. The virtual camera is“operated” (once again, virtually) at a given frame rate (or frames “persecond”). Thus, if the frame rate is 24, the virtual camera produces thefinal image 206 every second with 24 distinct frames in an imagesequence. Each frame comprising a number of 2D pixels. The pixels in aparticular frame are uniform in that they are of the same size. Anyindividual pixel in the volume dataset is sometimes referred to as avoxel.

Although it provides some basic interactivity, the “orbit” approachillustrated in FIG. 2 is quite limiting. FIG. 4 illustrates the approachof the subject disclosure, wherein the volume dataset 400 is renderedusing a virtual camera 402 that displays the final image 406 without anyposition constraint. In this approach, the virtual camera 402 may bemoved to any position outside or even within the volume dataset. Severalof these internal positions are illustrated. In this novel approach, thevirtual camera is movable (using the input device) inside the volumedataset, and the viewer can navigate along and through internalstructures that are represented therein. Thus, according to thisapproach, and unlike the prior art, the viewer has fullfreedom-of-motion within and around the volume dataset, or any portionthereof. This degree of interactivity provides a significantly enhancedand valuable user experience, as it provides much greater detail ofinternal structures. The technique allows the user to explore the humanbody non-invasively from every position and direction possible. Thevirtual camera (and thus the viewer) can move past tissue that mightotherwise block a view, and the camera can present views from inside thebody that were previously unattainable. In effect, the technique enablesthe viewer to see through multiple types of tissue simultaneously and toexamine the spatial relationship of different parts of the body.

There are two (2) techniques that facilitate the disclosed method: (i)continuous real-time dynamic rendering resolution, and (ii) continuousper pixel dynamic sampling distance for ray tracing volume datasets.Each of these techniques is now described. As used herein, “resolution”refers to a spatial number of pixels horizontally and vertically, withrespect to a picture (image) that is drawn from a particular displayframe. “Rendering” refers to a process by which the eventual picture isdrawn by the disclosed technique. In a representative embodiment,rendering is implemented by ray tracing, although this is not alimitation. The term “dynamic” refers to changes to the output renderingresolution at each frame, or as needed. The term “real-time” generallyrefers to a frame per second update rate greater than a predeterminedvalue, such as 24. The term “continuous” refers to the number of pixelsthat are added to or subtracted from a final picture every frame toensure that the picture only changes a small amount, to ensuresmoothness. The “continuous real-time dynamic rendering resolution”function changes a number of pixels horizontally and vertically by asmall amount in relation to a difference between a current frame rateand a desired frame rate, with respect to a picture that is drawn at aframe update rate (preferably >24 frames per second) to provide highresolution rendering. This feature is beneficial as it allows higherrendering quality than is possible for fixed resolution, which cannotguarantee real-time frame rates especially with respect to any positionin space.

The dynamic rendering resolution is illustrated in FIG. 5. This figureshows three (3) representative (random) frames of a set of displayframes of the displayed image. Each frame represents a single finalimage, and the set of final images (from the set of frames) representthe displayed image as rendered on the display at a given point in time.As illustrated, each pixel in each frame is uniform, although accordingto the “dynamic rendering resolution” function of this disclosure, theresolution can vary across particular frames. Thus, in this example,frame 1 as 16 pixels, frame 26 has 256 pixels, and frame 53 has 64pixels. Generalizing, when an image of the volume dataset is rendered ata given number of frames per second, at least two frames in the imagesequence have varying resolution.

This dynamic rendering resolution function preferably is achieved asfollows. Inside a main display processing loop, and at a minimum of“desired” frames per second, the routine calculates a difference betweena “current” frame rate and a “desired” frame rate. This frame ratedifference is what determines how the resolution will change for thisframe. When the difference is positive (i.e., when the desired framerate is greater than current frame rate), the display routine use oneless pixel column or pixel row alternately (or one less of each) in thefinal image to render a next frame. This operation “speeds up” therendering application and helps achieve the desired frame rate. If, onthe other hand, the difference in frame rate is negative (i.e., thedesired frame rate is less than the current frame rate), the displayroutine uses one more pixel column or pixel row alternately (or one moreof each) in the final image to render the next frame. This increases therendering resolution and, thus, the quality of the rendered image. Atthe end of each frame, the routine rescales the image back to screenresolution with or without interpolation to account for the change inthe number of pixels. This process speeds up the rendering because raytracing is inherently very dependent on the total number of cast rays inthe final image. If that number is reduced, the application speeds up.

In addition to dynamic rendering resolution, the display method of thisdisclosure implements an approach referred to as “continuous per pixeldynamic sampling distance for ray tracing,” as is now described. By wayof background, FIG. 6 illustrates a “fixed step” approach wherein thedistance from each previous sampling location (represented by a dot)along the length of the ray is the same. This fixed step pattern 600does not take into consideration the characteristics of the volumedataset itself (e.g., the density at the particular intersection pointin the CT scan, or the value of electron spin at the particular point inthe MRI scan, or some such similar value, depending on the type ofdata), but simply generates the sample at a particular uniform locationalong the ray. This fixed step approach does not produce satisfactoryresults. According to this disclosure, the sampling distance along thelength of a ray is permitted to vary dynamically within each pixel.Preferably, the step changes after a sampling of the dataset (and, inparticular, after each sampling) and in relation to a value contained inthe dataset. In particular, the values of this relationship preferablyare the value (e.g., density, for CT data) at the current voxel, and atarget or focus value (i.e., density of 750 Hounsfield units, which arestandard units for describing radio-density). The target or focus valuemay be user- or system-configurable. Further, preferably a distanceadded to or subtracted from a current step along the ray is small toensure smoothness as the ray traverses through the dataset. In apreferred embodiment, each ray (each corresponding to a pixel) adjustsits own unique step dynamically as it traverses through and samples thedataset. In effect, the ray speeds up or slows down as needed. For everypixel in the output image on a per sample basis, the display routineadjusts the unique sampling distance along the length of the ray castforward in space, which represents a single pixel in the rendered image,in relation to the values sampled from the dataset (i.e., an absolutevalue of a difference between the currently sampled value and the targetvalue). Generalizing, two pixels within a particular frame have avarying number of ray tracing steps relative to one another. Thisapproach facilitates high-resolution rendering at real-time frame rateswhile avoiding any restrictions on the virtual camera in terms of itsposition and orientation in space.

A preferred approach to implementing the per-pixel dynamic samplingdistance for ray tracing is now described. For every frame at real timerates, and for every pixel in the final image, the routine “starts” theray at the camera position. Then, the routine sets the ray's directionto be the camera direction plus the pixel position in the imagetransformed into world space. This operation amounts to an aperture orlens for the 3D camera; as a result, the ray has both a position and adirection. The program then steps down the ray, stopping at locations tosample the volume dataset. The distance that is stepped each frame isdependent on the value at the current sample point of the volume dataand a value (e.g., CT density, MRI electron spin, or equivalent) of thedesired tissue in current focus. Preferably, and as described above, thedistance stepped equals an absolute value of the difference between thecurrently sampled value (e.g., density) and the user- orsystem-configured target, multiplied by a small number to ensuresmoothness. In general, if the absolute value of the difference indesired tissue value and current sampled volume data is high, then alarger step is taken. If, however, the value of the difference indesired tissue value and current sampled volume data is small, then asmaller step is taken.

Preferably, and as illustrated in FIGS. 8-9, the process concentratesthe steps and samples in areas where the desired tissue values exist,while areas that are not in focus are spared (i.e., need not besubjected to dynamic stepping). At each step, preferably the routinetakes the scalar sampled volume data value and scales it (e.g., to therange of −1000 to 1000 standard Hounsfield units), and then the routineuses this value to look-up a color that corresponds to a certain type ofmaterial based on the Hounsfield number. (In the alternative, the valuecan be used directly for grey-scale images.) The routine then uses anaccumulation method (e.g., pixel color equals tissue differencemultiplied by the step color) to accumulate the color for this step ontothe pixel itself. Preferably, at each step, a small value is added tothe accumulated density. This process is repeated either until densityis greater than a desired value or until the step distance itself isvery small (e.g., a desired small value beyond which the eye can see),or until the maximum known depth is reached. When finished, the routinehas the final color for this pixel. Preferably, this approach is thenrepeated for every pixel in the final image.

When it is time for the next frame to be rendered, the camera is movedto its new position and orientation, and then process is repeated again.

For computational efficiency, the above-described approach may beimplemented using a GPU so that many pixels can be processed inparallel. In the alternative, a multi-core CPU can be used to facilitatethe parallel processing.

FIG. 10 illustrates a representative data processing system 1000 for usein processing the digital data in the above-described manner. A dataprocessing system 1000 suitable for storing and/or executing programcode will include at least one processor 1002 coupled directly orindirectly to memory elements through a system bus 1005. The memoryelements can include local memory 1004 employed during actual executionof the program code, bulk storage 1006, and cache memories 1008 thatprovide temporary storage of at least some program code to reduce thenumber of times code must be retrieved from bulk storage duringexecution. Input/output or I/O devices (including but not limited tokeyboard 1010, display 1012, pointing device 1014, etc.) can be coupledto the system either directly or through intervening I/O controllers1016. Network adapters 1018 may also be coupled to the system to enablethe data processing system to become coupled to other data processingsystems or devices through networks 1020. The rendering program thatimplements dynamic rendering resolution and dynamic per-pixel raytracing is stored in system local memory 1004, as are the datastructures and associated data generated during the rendering process.As noted above, in an alternative embodiment, the data processing systemincludes a GPU and associated graphics card components and, in thiscase, preferably the rendering program and volume data are stored ingraphics card memory.

While certain aspects or features have been described in the context ofa computer-based method or process, this is not a limitation of theinvention. Moreover, such computer-based methods may be implemented inan apparatus or system for performing the described operations, or as anadjunct to other dental restoration equipment, devices or systems. Thisapparatus may be specially constructed for the required purposes, or itmay comprise a general purpose computer selectively activated orreconfigured by a computer program stored in the computer. Such acomputer program may be stored in a computer readable storage medium,such as, but is not limited to, any type of disk including opticaldisks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs),random access memories (RAMs), magnetic or optical cards, or any type ofmedia suitable for storing electronic instructions, and each coupled toa computer system bus. The described functionality may also beimplemented in firmware, in an ASIC, or in any other known or developedprocessor-controlled device.

While the above describes a particular order of operations performed bycertain embodiments of the invention, it should be understood that suchorder is exemplary, as alternative embodiments may perform theoperations in a different order, combine certain operations, overlapcertain operations, or the like. References in the specification to agiven embodiment indicate that the embodiment described may include aparticular feature, structure, or characteristic, but every embodimentmay not necessarily include the particular feature, structure, orcharacteristic.

While given components of the system have been described separately, oneof ordinary skill will appreciate that some of the functions may becombined or shared in given systems, machines, devices, processes,instructions, program sequences, code portions, and the like.

The volume dataset may be generated from any data source. It is notrequired that the volume dataset be CT or MRI data, or that the dataitself be medical imaging data. The techniques herein may be used withinany volume dataset irrespective of content.

In one embodiment, a tangible (non-transitory) machine-readable mediumstores the computer program that performs the dynamic renderingresolution and dynamic per-pixel ray tracing during the process ofrendering the volume dataset on the display. The program receives thevolume dataset and renders the virtual camera construct (which livesinside the machine). The program moves and re-orients the camera underthe user's control, altering the view as desired. As described, thedynamic rendering resolution process increases or decreases the numberof pixels in each frame of a set of frames, while the per-pixel dynamicstepping increases or reduces the number of ray tracing steps (i.e.,step distance along the ray) per pixel. By continuously reducing theresolution across frames and reducing the number of steps (i.e., varyingthe step distance along the ray) per pixel within a frame, the programcan speed up its overall rendering of the image at the desired framerate, and in this manner the virtual camera construct can be positionedand oriented anywhere, including within the volume dataset itself. Thevirtual camera has complete freedom-of-motion within and about thevolume dataset; thus, the viewer has the ability to move to any positionin 3D space and look in any direction in real-time. The describedapproach enables real-time tissue selection and segmentation in 3D sothat various tissues (including bone) are visualized without requiringthe program to continually re-build a 3D mesh or to use preset tissuepalettes.

Having described our invention, what we now claim is as follows:
 1. Anarticle comprising a non-transitory machine-readable medium that storesa program, the program being executable by a machine having a hardwarecomponent and a display routine, comprising: program code to calculate,for a given frame of a display image, a difference between a currentframe rate and a desired frame rate and, based on the difference, torender a next frame of the display image by adjusting one of: a pixelcolumn, and a pixel row; and program code operative at an end of thegiven frame to rescale the display image back to a given screenresolution to account for a change in a number of pixels that resultsfrom adjustment of the pixel column or pixel row.
 2. The article asdescribed in claim 1 wherein when the difference is positive, thedisplay routine uses at least one less pixel column or pixel rowalternately to render the next frame of the display image.
 3. Thearticle as described in claim 1 wherein when the difference is positive,the display routine uses at least one less pixel column and pixel row torender the next frame of the display image.
 4. The article as describedin claim 1 wherein when the difference is negative, the display routineuses at least one more pixel column or pixel row alternatively to renderthe next frame of the display image.
 5. The article as described inclaim 1 wherein when the difference is negative, the display routineuses at least one more pixel column and pixel row to render the nextframe of the display image.
 6. The article as described in claim 1further including: program code to ray trace a volume dataset using aplurality of rays, each ray, as it traverses the volume dataset, samplesthe volume dataset along a set of sampling points whose locations alongthe ray are determined as the ray is cast, wherein, for each pair ofsampling points, a location of at least a next sample point along theray is determined by a value of a voxel at a current sample point, suchthat a distance between two sample locations varies dynamically. 7.Apparatus, comprising: a display; a processor; program code tocalculate, for a given frame of a display image, a difference between acurrent frame rate and a desired frame rate and, based on thedifference, to render a next frame of the display image by adjusting oneof: a pixel column, and a pixel row; and program code operative at anend of the given frame to rescale the display image back to a givenscreen resolution to account for a change in a number of pixels thatresults from adjustment of the pixel column or pixel row.
 8. Theapparatus as described in claim 7, further including: program code toray trace a volume dataset using a plurality of rays, each ray, as ittraverses the volume dataset, samples the volume dataset along a set ofsampling points whose locations along the ray are determined as the rayis cast, wherein, for each pair of sampling points, a location of atleast a next sample point along the ray is determined by a value of avoxel at a current sample point, such that a distance between two samplelocations varies dynamically.
 9. A display method, comprising:calculating, for a given frame of a display image, a difference betweena current frame rate and a desired frame rate and, based on thedifference, rendering a next frame of a display image by adjusting oneof: a pixel column, and a pixel row; and at an end of the given frame,rescaling the display image back to a given screen resolution to accountfor a change in a number of pixels that results from adjustment of thepixel column or pixel row; wherein at least one of the calculating andrescaling steps uses a hardware processor.
 10. The display method asdescribed in claim 9, further including ray tracing, using a hardwareprocessor, the volume dataset using a plurality of rays, each ray, as ittraverses the volume dataset, samples the volume dataset along a set ofsampling points whose locations along the ray are determined as the rayis cast, wherein, for each pair of sampling points, a location of atleast a next sample point along the ray is determined by a value of avoxel at a current sample point, such that a distance between two samplelocations varies dynamically.